
United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 

Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 22313-1450 
www.uspto.gov 



APPLICATION NO. 



FILING DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET NO. 



CONFIRMATION NO. 



09/938,256 



08/23/2001 



7590 



09/08/2005 



HEWLETT-PACKARD COMPANY 
Intellectual Property Administration 
P.O. Box 272400 
Fort Collins, CO 80527-2400 



Edward S. Beeman 



10003835-1 



1286 



EXAMINER 



EDWARDS, PATRICK L 



ART UNIT 



PAPER NUMBER 



2621 

DATE MAILED: 09/08/2005 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 10/03) 




United States Patent and Trademark Office 



Commissioner for Patents 
United States Patent and Trademark Office 
P.O. Box 1450 
Alexandria, VA 22313-1450 

www.uspto.gov 



BEFORE THE BOARD OF PATENT APPEALS 
AND INTERFERENCES 



MAILED 

SEP 0 8 2005 

Technology Center 2600 

David R. Risley 
For Appellant 

EXAMINER'S ANSWER 



Application Number: 09/938,256 
Filing Date: August 23, 2001 
Appellant(s): BEEMAN ET AL. 



This is in response to the appeal brief filed 27 June 2005 



Application/Control Number: 09/938,256 Page 2 

Art Unit: 2621 

(1) Real Party in Interest 

A statement identifying by name the real party in interest is contained in the brief. 

(2) Related Appeals and Interferences 

The examiner is not aware of any related appeals, interferences, or judicial proceedings which will directly affect or 
be directly affected by or have a bearing on the Board's decision in the pending appeal. 

(3) Status of Claims 

The statement of the status of claims contained in the brief is correct. 

(4) Status of Amendments After Final 

The appellant's statement of the status of amendments after final rejection contained in the brief is correct. 

(5) Summary of Claimed Subject Matter 

The summary of claimed subject matter contained in the brief is correct. 

(6) Grounds of Rejection to be Reviewed on Appeal 

The appellant's statement of the grounds of rejection to be reviewed on appeal is substantially correct. The changes 
are as follows: 

WITHDRAWN REJECTIONS 

The following grounds of rejection are not presented for review on appeal because they have been withdrawn by the 
examiner. The rejections under 35 USC 112, Second Paragraph to claims 9, 10, 21, 22, and 24-27 are withdrawn. 
This was previously stated in item (6) above. 

• Appellant's brief presents arguments relating to improper final status of rejection on pages 6-7 of the appeal 
brief (see subheading (A)(2)). This issue relates to petitionable subject matter under 37 CFR 1.181 and not to 
appealable subject matter. See MPEP § 1002 and § 1201. 
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(7) Claims Appendix 

The copy of the appealed claims contained in the Appendix to the brief is correct. 

(8) Evidence Relied Upon 

The following is a listing of the evidence (e.g., patents, publications, Official Notice, and admitted prior art) relied 
upon in the rejection of claims under appeal. 
■ Kagami et al. (USPN 5,974,422) 
- Zhuetal. (USPN 6,345,274) 

(9) Grounds of Rejection 

The following ground(s) of rejection are applicable to the appealed claims: 

1. Claims 1 and 3-27 have been rejected under 35 USC 103(a) as being unpatentable over the combination of 
Zhu et al. (USPN 6,345,274) and Kagami et al. (USPN 5,974,422). This rejection is provided directly below. 

Claim Rejections - 35 USC § 103 

2. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set 
forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

3. Claims 1-27 are rejected under 35 U.S.C. 103(a) as being unpatentable over the combination of Zhu et al. 
(USPN 6,345,274) (hereinafter 'Zhu') and Kagami et al. (USPN 5,974,422). 

With regard to claim 1, Zhu discloses querying a user as to the attributes of an image the user wishes to 
retrieve (Zhu col. 6 lines 8-9). Since the images are measured and identified based on their features (Zhu col. 5 lines 
14-24 and col. 6 lines 15-33), which are determined by image attributes (Zhu col. 5 lines 34-36 and col. 6 lines 36- 
44 in conjunction with Figure 3). It follows that when a user selects a desired image, the user is inherently 
identifying the attributes of that image. 

Zhu further discloses receiving user responses and presenting images to the user based upon these user 
responses (Zhu col. 6 lines 10-14). Zhu discloses that the user selects 'desired response images' from the initial 
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query. These selections, (which are received by the system), are the user responses to the query. Furthermore, these 
selections identify the image attributes that the user wishes to retrieve. 

Zhu further discloses presenting images to the user based upon these responses (Zhu col 6 lines 14-18 and 
col. 7 lines 30-3 1 in conjunction with Figure 7). These images are presented to the user via the display 30 disclosed 
in Zhu. 

Zhu fails to expressly disclose querying the user by posing a series of explicit questions to the user, and 
then receiving explicit user responses to these questions. As was discussed in the previous office action, the user 
query disclosed in Zhu consists of implicit questions. 

The examiner recognizes that the images of this response set do not come with a question mark. But this is 
not a requirement. A question need not be stated expressly and with accompanying punctuation. When a computer 
user is presented a set of several images, and selects desired images from this set, this is an implicit question. The 
system is, in essence, asking the user to provided examples/counterexamples (+/- examples) to further facilitate 
image retrieval. 

This situation often occurs in computing. A user is prompted to make a selection, and the user makes a 
selection at that prompt. In many computing situations such as these, a prompt such as this will not come with a 
question mark. However, users of computers realize that their next move is to input a response. Thus, the system is 
implicitly asking them a question. It is implicitly requesting that they make response. 

Accordingly, Zhu discloses the limitation of presenting questions to a user, but fails to expressly disclose 
that these questions are explicity. Kagami, however, discloses posing a series of explicit questions to users, and then 
receiving explicit user responses (Kagami col. 3 lines 1-1 1). It would have been obvious to one reasonably skilled 
in the art at the time of the invention to modify Zhu's image retrieval method by using explicit questions as the 
mechanism of querying a user as taught by Kagami. Such a modification would have allowed for a user friendly 
method of user/system interaction and would have resulted in a more precise analysis of the user's appraisal of a 
given image (Kagami col. 3 lines 10-1 1). 

With regard to claim 3, Kagami discloses that the successive questions are dependent on the user response 
given to a previous question (see kagami col. 9 line 49 - col. 10 line 14). 

With regard to claim 4, Zhu discloses eliminating potential image matches in response to the received 
response (Zhu col. 6 lines 13-21). Zhu discloses retrieving images that are similar to the selected image. 
Consequentiy, the potential matches which are different from the selected image are eliminated. This limitation is 
further disclosed in Kagami (see Kagami col. 9 line 49 - col. 10 line 14). 

With regard to claims 5, 7 and 23 Kagami discloses explicitly selecting a portion of an image presented to 
the user (Kagami col. 7 lines 24-28: The refernce describes that the user selects a portion (i.e. tops or bottoms) of 
the presented image. It follows that the selected portions qualify as portions of an image, and as images in and of 
themselve). 

With regard to claims 6 and 8, Zhu further discloses that the image features are stored as image metadata 
(col. 5 lines 14-16). Since the image "features" disclosed in Zhu are determined by the attribute values of the image 
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(see Figure 3 of Zhu), it follows that the attribute values are inherently stored as image metadata. Additionally it 
should be noted that this occurs in response to the user selection of an image as disclosed in Zhu (i.e. the claimed 
'user response to a prompt'). This is shown in col. 6 lines 13-41 in the Zhu reference. Furthermore, these 
limitations are disclosed in the Kagami specification. Kagami teaches a method for creating and storing 'kansei 
information', which is analogous to the 'image metadata' recited in the claim. 

With regard to claim 9 and 10, Zhu discloses extracting recognizable image attributes from the selected 
images (Zhu col. 5 lines 14-36 and col. 6 lines 13-3 1 in conjunction with Figure 3). It was argued above that Zhu 
inherently determines attribute values in the disclosed feature extraction step. This feature extraction step qualifies 
as the 'image analysis' recited in the claim in that an image must first be analyzed to determine its features before 
they can be extracted. Zhu further discloses that this is performed on "recognizable image attributes" such as color, 
texture, regions, boundaries, etc. (col. 5 lines 18-20). Further, Zhu teaches that this analyzing process is during an 
image storage process. Figures 2 and 7, for example, show that the analyzing of images for a recognizable image 
attribute is performed during an image storing process. For instance, Fig. 7 shows that the candidate set — which is 
determined by analyzing images for a recognizable image attribute — is saved (or stored).). Indeed, the entire 
disclosure of both references is concerned with storing and retrieving of images. 

With regard to claims 21 and 22, Kagami further discloses prompting the user for keywords or phrases 
relevant to image content, and then storing the keywords as metadata (Kagami col 7 lines 38-58 in conjunction with 
Figures 7a-b). 

Referring to claims 11-15 and 24-25, which merely add a system for performing the method discussed 
above, the prior art discloses such a system (see Figure 1 of Zhu and Figure 2 of Kagami). 

Referring to claims 16-20 and 26-27, which merely add a computer program for performing the method 
discussed above, Zhu discloses such a computer program (see Zhu col. 4 lines 19-25). 

NEW GROUND (S) OF REJECTION 
Claim Rejections - 35 USC §102 

4. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the 
rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on 
sale in this country, more than one year prior to the date of application for patent in the United States. 

5. Claims 1-27 are rejected under 35 U.S.C. 102(b) as being clearly anticipated by Kagami et al. (USPN 
5,974,422) 

As applied to claim 1, Kagami discloses querying a user as to at least one attribute of an image the user 
wishes to retrieve by posing a series of explicit questions to the user (Kagami col. 3 lines 1-11: The reference 
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describes posing a series of explicit questions to the user that are in regards to whether the user likes certain 
aspects/attributes of the image.). 

Kagami further discloses receiving explicit user responses to the posed questions (Kagami col. 10 lines 1-9: 
The reference describes several different types of user responses, such as a simple e yes' or 'no\ a rating of the item, 
or a specific user input requirement. All of these responses are explicit). 

Kagami further discloses presenting at least one image to the user based upon the user responses (Kagami 
Fig. 6 element 603 in conjunction with col. 9 lines 56-67: The reference describes outputting/presenting an image to 
a user based on the user's kansei. A user's kansei, of course, is based on their explicit responses (see col. 1 lines 23- 
29).). 

As applied to claim 3, Kagami further discloses that the successive questions depend upon the user 
response given to a previous question (Kagami Figs. 15a and 15b in conjunction with col. 9 lines 50-67: A different 
image/question will be given in figure 15b dependent on the answer provided by the user in 15a. Further, this 
limitation is generally disclosed by the learning system of kagami in that the images/questions presented to the user 
are dependent on a score which is computed from responses to previous questions.) 

As applied to claim 4, Kagami further discloses eliminating potential images matches in response to the 
received responses (Kagami col. 5 lines 50-65: The reference describes that user responses generate criterion by 
which future objects are judged. The disclosed example is of a system for making recommendations about women's 
clothing. Depending on the score derived from the user's selections, the system presents the desired item of 
clothing. For example, if the system determines that the user desireth pantalones, then skirts are thereby eliminated 
as a potential image match.). 

As applied to claim 5, Kagami discloses prompting the user to explicitly identify an image attribute of an 
image presented to the user so as to increase the proficiency with which images are retrieved for the user (Kagami 
col. 10 lines 1-9: The reference gives the example of the user response "I want to make the image reddish in 
general." This increases the retrieval proficiency because only the colors red, crimson and orange will be retrieved. 
Similarly, the user could identify types of clothing. For instance, a user could explicitly specify that she desires a 
certain style of pantalones, skirt, or blouse.). 

As applied to claim 6, Kagami discloses storing image metadata in response to the user identification, the 
metadata identifying the image as containing the image attribute that the user identified (Kagami col. 10 lines 1-9 
and col. 9 lines 37-46: These two passages describe that a user identified attribute is stored. This user identifed 
attribute is metadata, since it is data about data.). 

As applied to claim 23, Kagami discloses explicitly identifying at least one image attribute comprises 
explicitly selecting a portion of an image presented to the user (Kagami Figs. 15a and 15b: The figures show that 
the user selects portions of the image with a cursor.). 

As applied to claim 7, Kagami discloses prompting the user to identify select images of the images 
presented to the user that each contains a particular image attribute so as to increase the proficiency with which 
images are retrieved for the user (Kagami figure 14 and col. 9 lines 25-35). 
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As applied to claim 8, Kagami discloses storing image metadata in response to the user identification, the 
metadata identifying the select images as each containing the particular image attribute (Kagami figure 14 and col. 9 
lines 25-35: The user identification is stored, and this information — as is discussed above — is metadata.). 

As applied to claim 9, Kagami discloses analyzing images for a recognizable image attribute during an 
image storing process (Kagami col. 5 lines 42-50: The reference describes that during an image storing process, an 
input image is divided into structural elements. This process of division into structural elements entails analyzing 
images for a recognizable attribute. The structural elements are parts of the design image (i.e. different types of 
clothing) and dividing an image into structural elements involves analyzing the images to break them apart into 
structural elements such as skirt, v-neck, etc. (see further col. 8 lines 22-30).). 

As applied to claim 10, Kagami discloses storing image metadata in response to the analyzing, the metadata 
identifying an analyzed image as containing the recognizable image attribute (Kagami col. 8 lines 22-30 in 
conjunction with Fig. 10: The reference describes and shows the storing the data about the structural elements. This 
data is metadata since it is data about the data displayed to a user.). 

As applied to claim 21, Kagami discloses prompting the user for keywords or phrases during an image 
storing process, the keywords or phrases being relevant to content of an image (Kagami col. 10 lines 1-9). 

As applied to claim 22, Kagami discloses storing keywords as metadata in response to receving keywords 
or phrases by the user, the metadata identifying the image as containing content described by the keywrods or 
phrases (Kagami col. 10 lines 1-9: Again, the information provided by the user is metadata.). 

As applied to claims 11-15 and 24-25, all of the limitations of the claims have been addressed above, 
except that that these claims are directed to a means for performing the method, isntead of the method itself. 
Kagami discloses such a means e.g. at Figs. 1 and 2. 

As applied to claims 16-20, and 26-27, all of the limitations have been addressed above, except that these 
claims are directed to a computer program comprising logic configured to perform the method, instead of the 
method itself. Kagami discloses a computer system (Kagami Figs. 1 and 2). A logic configured program is inherent 
in such a computer system. 

(10) Response to Argument 

(a) Applicant alleges that Zhu fails to teach querying a user by posing questions to the user. Specifically, applicant 
alleges that Zhu does not query the user at all, but that the user queries the system (appeal brief pgs. 11-12). 
(response) The examiner disagrees. Zhu discloses that a default query is performed to generate a response set. 
From this generated response set, the user selects examples or counterexamples (Zhu col. 6 lines 9-12: This 
passage — in conjunction with Fig. 7 — shows that the user specifies +/- examples from a response set. Thus, the 
response set as presented to the user is an implicit question urging the user to select desired response images from 
the response set.). 
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The examiner recognizes that the images of this response set do not come with a question mark. This is not 
needed. When a computer user is presented a set of several images, and selects desired images from this set, this is 
an implicit question. The system is, in essence, asking the user to provided examples/counterexamples (+/- 
examples) to further facilitate image retrieval. 

This situation often occurs in computing. A user is prompted to make a selection, and the user makes a 
selection at that prompt. In many computing situations such as these, a prompt such as this will not come with a 
question mark. However, users of computers realize that their next move is to input a response. Thus, the system is 
implicitly asking them a question. It is implicitly requesting that they make response. 

In applicant's brief, it was stated that applicant was unaware of what an implicit question is. The examiner 
has used the term "implicit question" simply because it is not expressly stated. Nonetheless, it is still a question. A 
complete sentence with an accompanying question mark is not a prerequisite for something to be defined as a 
question. It is the examiner's position that the Zhu system is posing a question to the user notwithstanding the 
absence of a worded sentence and/or punctuation. 

For example, a computer system can display two images and have any of: 

(1) a sentence phrased as a question such as "which image do you prefer?"; or 

(2) a descriptive sentence such as "select the preferred image,"; or 

(3) each image might have an associated radio button that the user could use to select the preferred image. 
All three of these situations are exactly equivalent. The first situation presents a formal question. The second 
situation uses words to prompt the user, but is not phrased in the form of a proper question. The third situation does 
not even use words to prompt the user. Given applicant's desired interpretation, only the first situation would be 
considered a question, even though the same result is reached in all three situations. The examiner disagrees with 
this narrow interpretation of the word "question" and respectfully submits that all three situations represent the 
posing of a question to the user. Similarly, the Zhu system discloses posing a question to the user. 

(b) Applicant alleges that the combination of Zhu and Kagami is improper. Specifically, applicant states that that 
because Zhu does not pose implicit questions, that "there would be no motivation to a person having ordinary skill 
in the art to replace Zhu's supposed "implicit questions" with Kagami's explicity questions (appeal brief pg. 12) 
Therefore, Applicant's argument is totally reliant on the above (a) argument. 

(response) The examiner disagrees. Applicant's argument hinges upon the above part (a) argument. As was stated 
in the response to part (a), Zhu discloses presenting questions to a user, but does not expressly disclose that these 
questions are stated explicitly. Kagami cures this deficiency by disclosing that explicit questions are presented to 
the user. For these reasons, the references are easily combinable, and when combined, all the limitations of the 
claim are met. 
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(c) Regarding dependent claims 9, 14, and 19, Applicant alleges that "the Examiner's reluctance to consider the 
limitation stemmed from either a lack of prior art teachings to meet the expressed claim limitation or reluctance to 
conduct a further prior art search. . . (appeal brief pgs. 7-8). 

(response) The examiner disagrees. The examiner didn't expressly consider the limitation because of a good faith 
belief that the limitation was not in accord with the parent limitations. The examiner has since been persuaded that 
these limitations can be interpreted can be reconciled. The limitation in question, however, is taught by the 
combination of Zhu and Kagami. Indeed, this limitation is conspicuously taught by both Zhu and Kagami. The 
relevance of Kagami to this limitation is provided above in the 102b rejection with Kagami. 

Further, this limitation is disclosed in Zhu, as can be seen e.g. in Figs. 2 and 7 (The figures show that the 
analyzing of images for a recognizable image attribute is performed during an image storing process. For instance, 
Fig. 7 shows that the candidate set — which is determined by analyzing images for a recognizable image attribute — is 
saved (or stored).). Indeed, the entire disclosure of both references is concerned with storing an retrieving of 
images. 

(11) Related Proceeding(s) Appendix 

No decision rendered by a court or the Board is identified by the examiner in the Related Appeals and Interferences 
section of this examiner's answer. 



Conclusion 

6. For the above reasons, it is believed that the rejections should be sustained. 

7. This examiner's answer contains a new ground of rejection set forth in section (9) above. Accordingly, 

appellant must within TWO MONTHS from the date of this answer exercise one of the following two options to 

avoid sua sponte dismissal of the appeal as to the claims subject to the new ground of rejection: 

(1) Reopen prosecution. Request that prosecution be reopened before the primary examiner by filing a reply 
under 37 CFR 1.111 with or without amendment, affidavit or other evidence. Any amendment, affidavit or 
other evidence must be relevant to the new grounds of rejection. A request that complies with 37 CFR 
4 1 .39(b)(1) will be entered and considered. Any request that prosecution be reopened will be treated as a 
request to withdraw the appeal. 
(2) Maintain appeal. Request that the appeal be maintained by filing a reply brief as set forth in 37 CFR 41.41. 

8. Such a reply brief must address each new ground of rejection as set forth in 37 CFR 41.37(c)(l)(vii) and 
should be incompliance with the other requirements of 37 CFR 41.37(c). If a reply brief filed pursuant to 37 CFR 
4 1 .39(b)(2) is accompanied by any amendment, affidavit or other evidence, it shall be treated as a request that 
prosecution be reopened before the primary examiner under 37 CFR 41.39(b)(1). 
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Extensions of time under 37 CFR 1 .136(a) are not applicable to the TWO MONTH time period set forth 
above. See 37 CFR 1.136(b) for extensions of time to reply for patent applications and 37 CFR 1.550^) for 
extensions of time to reply for ex parte reexamination proceedings. 



Respectfully submitted, 
Patrick L. Edwards 






ANDREW W.JCHNS 
PRIMARY EXAMINER 



9. A Technology Center Director or designee must personally 
approve the new ground (s) of rejection set forth in section (9) 
above by signing below: 

Leo H. Boudreau 




Conferees: 
Joseph 

Brian We^rfer 
Andrei Johns 




